DOCUMENT RESUME 



ED 387 504 



TM 023 662 



AUTHOR 
TITLE 



PUB DATE 
NOTE 



PUB TYPE 



Stake, Robert; Migotsky, Christopher 

Evaluation Studies of the Chicago Teachers Academy: 

Methods and Findings of the CIRCE Internal Evaluation 

Study. 

Apr 95 

13p.; Paper presented at the Annual Meeting of the 
American Educational Research Association (San 
Franciaco, CA, April 18-22, 1995). 
Reports - Evaluative/Feasibility (142) -- 
Speeches/Conference Papers (150) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MFOl/PCOl Plus Postage. 

"Educational Improvement; Elementary Secondary 
Education; Evaluation Methods; "Inservice Teacher 
Education; Interviews; Mathematics; Principals; 
•'Program Evaluation; Qualitative Research; Sciences; 
•'Staff Development; Teacher Attitudes; ''Teachers 
•'Chicago Teachers Academy IL; Internal Evaluation 



ABSTRACT 



The Center for Instructional Research and Curriculum 



Evaluation (CIRCE) was commissioned by the Chicago (Illinois) 
Teachers Academy for Mathematics and Science to conduct an internal 
evaluation oC their programs. Academy managers wanted documentation 
of Academy efforts in the schools that they served. The CIRCE team 
left it to other evaluation teams to measure impact through 
standardized testing, the use of preordinate templates, and the 
analysis of large-scale distribution data, concentrating on a 
responsive evaluation plan with comprehensive, even personal ist ic , 
observation of Academy impact in unique situations. Qualitative 
observations, interviews with 20 teachers and 20 principals at the 42 
schools the Academy serves, and document review were used to explore 
key issues of teacher staff development and school improvement. The 
findings of this interpretive study show the Academy to have been 
providing ■ arefully selected schools with good quality content and 
pedagogical assistance. The burden on teachers, however, in terms of 
time and effort, even of well-received assistance programs is 
detailed. The Acadsmy contribution was significant and valued by the 
communities served. Two handouts supplement the discussion. 
(Author/SLD) 
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CIRCE Internal Evaluation 2 

Abstract 

The Center for Instructional Research and Curriculum Evaluation (CIRCE) was 
commissioned by the Teachers Academy for Mathematics and Science to conduct an 
internal evaluation of their programs. Academy managers wanted documentation of 
Academy effects in the schools that they served. The CIRCE team left to NCISE and 
other evaluation teams the measuring of impact with standardized testing, use of 
preordinate templates and analysis of large scale distribution data; instead focusing on a 
responsive evaluation plan with comprehensive, even personalistic, observation of 
Academy impact in unique situations. Qualitative observations, interviews, and document 
review were used to explore key issues of teacher staff development and school 
improvement. The findings of this interpretive study show the Academy to have been 
providing carefully selected schools with good quality content and pedagogical assistance. 
The burden on teachers of well-received assistance programs was detailed. The Academy 
contribution was significant and valued by the communities served. 
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CIRCE Internal Evaluation 3 

In the summer of 1993, the Teachers Academy contracted CIRCE to conduct a 
year-long evaluative review of operations in an effort to interpret the quality of staff 
development and school improvement offerings. Our plan was heavy on process and 
issues, weak on distributions of outcomes, because we were skeptical about the validity of 
standardized instruments for capturing the effectiveness of Academy work. But, through 
intense scrutiny of that work in situation after situation, anticipated we would be able to 
interpret much about Academy impact and quality. The evaluation was different from 
many because it favored issues as conceptual structure rather than goals and promises of 
the organization— with most issues having less a product orientation, more one of process. 
Our foreshadowing questions are shown on the first handout sheet: 

Handout #1 



Descriptive 

• How does the Academy interact with the leadership of the Chicago schools and other 
parties across the city in order to effect a renewal of administiation, teaching, and 
learning? 

• What is the nature of Academy follow-up activity and what more is needed? Is 
CANAL part of the picture? What other staff development initiatives are underway? 

• How has the loss of many senior teachers and principals, partly through "Five plus 
Five" retirement, changed the picture of staff development? How has the Academy 
responded? 

• How do the NCTM Standards fit into re-development of math teaching at 
participating schools? 

Causal 

• How has teacher knowledge, pedagogy, and belief been affected by participation in 
Academy activities? 

Evaluative 

• How do the strategies and procedures for Academy deliveiy of service meet the needs 
of teachers? 

• What are the advantages and disadvantages of MathTools and Teaching of Integrated 
Math and Science (TIMS) for staff development? 

• Do present student achievement testing and assessment complement or deflect from 
Academy efforts? 
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In the style of responsive evaluation, these carefully-tailored descriptive, causal, 
and evaluative questions served to organize our data collection. But they were expected 
to be refined further as activities provided us with a firm sense of the work of the 
Academy— new issues might emerge and priorities would certainly change. By mid-study, 
the issue questions had become those listed on the top of the second handout: 

Handout #2-Top 



Descriptive/Interpretive 

• Were Academy activities for participating teachers consistent with state-of-the-art 
pedagogical emphases on experiential learning and local school decision-making? 

• Was the District emphasis on standardized achievement testing consistent with 
Academy emphasis on TIMS and MathToois? 

• Was there integration of mathematics and science in the participating schools? 

• In math instruction in participating schools, did we find more hands-on experiences, 
cooperative learning, problem-conceptualizing? 

• How did the Academy, Chicago Public Schools, universities and others resolve 
difference^i about staff development and school improven^kent? 

• Were there problems for a school working with several assistance groups at the same 
time? 

• What position did the Academy take on Local School Councils assuming greater 
responsibility for teacher selection and retention? 

• Was the Academy a key component in city and state educational initiatives? 

• What was the nature of Academy leadership activity and contribution to policy setting 
in Chicago school reform? 

Causal 

• Did emphasis on problem-solving and experiential learning make classrooms appear to 
lack seriousness of purpose and irritate people who want strong discipline in the schools? 

Evaluative 

• Were MathToois and TIMS too abstract for teachers and students at participating 
schools? 

• Were the schools and teachers most needing help the ones participating? 

• Was the Academy working to diminish professional mores that keep teachers from 
acknowledging weaknesses and trying to collectively to remediate poor teaching? 

• Did the Academy get good support from principals of participating schools? 

• How well did Academy teaching ref->rm strategy deal with teacher turnover? 
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It should be seen that the resulting data gathering aimed less at inventorying 
accomplishment than at illustrating instances of good and poor service delivered. Our 
report is carried by our interpretations, drawn partly from our experience and points of 
view. In some instances, to some people, they appear impressionistic~they are, but if we 
have done the work well, they are thoroughly worked impressions drawn upon numerous 
perspectives from both inside and outside the evaluation team. It is the intent of 
qualitative research to make assertions rooted in prior comprehension as well as data. 

Methods. The final report was due August 15, eight months ago, following eight 
months of observation. We had designed it to include case studies of three 
"implementation phase" elementary schools, selected mainly in terms of our opportunity to 
learn about staff development and the use of MathTools and TIMS in context. Each case 
school got a minimum of ten days on site, with burden to the schools held in check. At 
the Academy's 42 schools we interviewed 20 teachers and 20 principals, randomly 
selected, with an emphasis on their perceptions of Academy impact, using standardized 
and open-ended probe questions. And we did a document review of all participant School 
Improvement Plans'(SIPs) looking for changes over time reflecting Academy influence. 
Within two months, when it became apparent that we could not understand enough of 
what was happening without extensive review of Academy staff development activities 
and allied work of its new School Improvement Unit, we negotiated with Academy 
management to extend the scope of our work in those directions, and subsequently ran 
into the NCISE evaluation team more frequently. 
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General Findings. We found that the Teachers Academy for Mathematics and 
Science had made major contributions to the teaching performance of teachers in 
participating elementary schools in the public school system of Chicago. According to our 
observations, staff members at the Academy had developed a program of workshop 
experiences which demonstrated teaching methods appropriate for a sometimes brutal, 
changing society with a large proportion of its people greatly in need of custodial care. 
Participants finishing a series of workshops knew quite a bit more about what teaching 
should be. Educational goals had been proclaimed by the State of Illinois and several 
professional organizations. How to attain these fine goals remained a conjecture for this 
population. To lay the beginnings of a pathway to those goals, the Academy drew from 
numerous state-of-the-art curriculum developments to create a sound teacher development 
program of its own. 

We interpreted the Academy's path as conceptually sound and practical. In 
schoolrooms of Academy participating teachers, there appeared to be a small gradual 
change in the role of the student, a role consistent with the problem-solving, hands-on, 
constructivist pedagogy of leading science and mathematics educators. Student talk was 
becoming increasingly sophisticated, particularly in mathematics. Students were able to 
describe the tasks they faced, to recognize situations in which there was too little 
information given, to discuss inappropriate protocols followed in getting wrong answers 
and the ways that different students used different protocols, to recognize patterns of 
problems as similar despite different contexts, e.g., science and language aits. These 
students were able to take both an external and internal view of problem solving. 
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The normal road to betterment—that adopted by many change experts— is a 
corrective strategy termed "rational restructuring." Goal-setting, prioritizing, planning, 
setting milestones, evaluation, maximizing formalization and explication are all common 
components of a rational management system. The Academy didn't place great reliance 
on formal collection of management information. This tendency proved bothersome to the 
CIRCE team, exasperating to the NCISE team. Yet, there was reason to doubt that the 
Academy would perform its staff development and school improvement responsibilities 
better if it tried to become more rational in its operations. We viewed its style to be 
intuitive, operating reflexively from personal experience— particularly with sensitivity to 
the cultural and political contexts of city and school activities. To say that a style is 
intuitive is not to say it is faulty. The style of many outstanding political and business 
leaders, athletes, and scientists is largely intuitive. Intuitive management can be good or 
bad, just as rational management can be good or bad. Bureaucratic management is often 
ineffective because it is too rational, fixed upon rules and justifications, too little on 
personal understanding and common sense. Matters at the Academy were more talked 
about than written down, remembered more than filed. This organizational style allowed 
the Academy to adapt to uniqueness in schools and fast changing needs, but perhaps less 
able to unite on a common platform of purpose and operation. This intuitive style is 
generally common in schools and certainly pervasive throughout the Chicago public 
school system. Thus, the Teachers Academy was operating within a framework that was 
consistent with the surrounding components— if they had relied heavily on formalisms, the 
Academy's ability to articulate with other agencies could have been hobbled. 
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The Particulars. Each school in Chicago sees itself as a special case with special 
obstacles, as it should. Our tliree case study schools were no different; each had its unique 
story to tell. Each story had common elements as well, but we chose not to compare 
them. All three case study schools were participating in multiple programs for enriching 
education, some of those programs offered staff development for teachers. But for most 
teachers the Academy was the primary external influence. 

Key successful features of the Academy program were experiential training of 
teachers, provision of classroom materials that teachers knew how to use as a result of 
their training, and extended classroom support and follow-up by Academy personnel. One 
first grade teacher, Lisa Smith, spoke for many as she described her experience: 
At the workshops, they gave us the materials and the activities and let us 
experiment ourselves with the stuff, just like the kids would. There wasn't much 
talk about cooperative learning in the workshops, but we worked in groups at 
stations and that causes cooperative learning almost automatically. They probably 
had it in mind. We talked and shared answers. The instructors didn't say "Don't 
do that." One good activity was on Area, where you take square inch tiles and 
[measure an irregular] space. Kids like that activity. We drew dinosaurs outside 
and measured them with meter sticks. We did it on a day when Jeannie Hall from 
[the Academy] was there. She would always come in the morning and ask what 
kind of help I would need. I really miss her. If Jeannie hadn't come, maybe the 
kids wouldn't have gotten the flill idea of my knowledge. I might have tried a lot 
of the activities, but she added enrichment. She asked the kids questions I 
wouldn't have thought of She was more scientific. Now I can do more of what 
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she did. I had the chance to observe her and learn from her successes and her 
failures. Not just that I do exactly what she did. I learned from observations and 
experience more than from her telling me things. The school-wide activities, like 
the project on estimating and the one on height and arm length, brought us 
together. 

Despite appreciation for the Academy workshops, being out of the classroom was 
an often noted concern from participating teachers. As one teacher stated, "I'm a noted 
crab. Even though I'm not the children's favorite person, they would ask me not to go to 
the Academy workshops because they hated having a sub that often." Stability in the 
classroom was fragile and hard-won, something that teachers were cautious to disrupt 
with excursions to Academy workshops. 

In addition to Academy endeavors, other assistance programs called for teacher 
attentions. One in-school facilitator commented on the number of help programs working 
within her school. "Such a burden. Each program needs committees, and a teacher in 
charge of the effort here, and hospitality there. Each comes with a gift, but each extracts a 
price, such as workshops, liaison duties,..." CIRCE evaluators recognized that the schools 
could be better, that many were trying to lend a helping hand. The load these schools 
could bear was not adequately being assessed by the Academy or other reform groups 
within Chicago. They were not helping school people decide, gi\'e.ii > , resources 
available, what responsibilities to diminish, what were attainable goals and standards. 

In the end it was not the issue questions on your handout sheets, but the issue 
topics listed last there that structured the CIRCE evaluation report. 
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Handout #2--Bottom 

• The burden on teachers of providing them assistance. 

• Lack of agreement on goals for mathematics and science. 

• Misdirection of strategic planning. 

• The insuitability of science teaching packages. 

Ve worked hard to have a strongly interpretive character in the CIRCE report, without 
direct attention to immediate decision-making, without recommendations. In the field, we 
looked especially for the unexpected event, the telling moment, within our issue 
framework. We looked for others, triangulated. Sometimes only a few instances led 
directly to final report assertions. To find meaning and value, aggregation was not as 
important as understanding in context. We started with good questions and dug deeply 
into program activity. What we provided to the Academy was an admiring but problem 
beset portrayal, something of a persuasion to others that Academy operations were 
reasonably on target, but falling short of hope and expectations. 



ERIC 



ll 



Handout #1 



Foreshadowing Questions 

Descriptive 

• How does the Academy interact with the leadership of the Chicago schools and other 
parties across the city in order to effect a renewal of administration, teaching, and 
learning? 

• What is the nature of Academy follow-up activity and what more is needed? Is 
CANAL part of the picture? What other staff development initiatives are underway? 

• How has the loss of many senior teachers and principals, partly through "Five plus 
Five" retirement, changed the picture of staff development? How has the Academy 
responded? 

• How do the NCTM Standards fit into re-development of math teaching at 
participating schools? 

Causal 

• How has teacher knowledge, pedagogy, and belief been affected by participation in 
Academy activities? 

Evaluative 

• How do the strategies and procedures for Academy delivery of service meet the needs 
of teachers? 

• What are the advantages and disadvantages of MathTools and Teaching of Integrated 
Ma h and Science (TIMS) for staflF development? 

• Do present student achievement testing and assessment complement or deflect from 
Academy efforts? 
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Evaluation Issue Questions 
Descriptive/Interpretive 

• Were Academy activities for participating teachers consistent with state-of-the-art 
pedagogical emphases on experiential learning and local school decision-making? 

• Was the District emphasis on standardized achievement testing consistent with 
Academy emphasis on TIMS and MathTools? 

• Was there integration of mathematics and science in the participating schools? 

• In math instruction in participating schools, did we find more hands-on experiences, 
cooperative learning, problem-conceptualizing? 

• How did the Academy, Chicago Public Schools, universities and others resolve 
differences about staff development and school improvement? 

• Were there problems for a school working with several assistance groups at the same 
time? 

• What position did the Academy take on Local School Councils as?uming greater 
responsibility for teacher selection and retention? 

• Was the Academy a key component in city and state educational initiatives? 

• What was the nature of Academy leadership activity and contribution to policy setting 
in Chicago school reform? 

Causal 

• Did emphasis on problem-solving and experiential learning make classrooms appear to 
lack seriousness of purpose and irritate people who want strong discipline in the schools? 

Evaluative 

• Were MathTools and TIMS too abstract for teachers and students at participating 
schools? 

• Were the schools and teachers most needing help the ones participating? 

• Was the Academy working to diminish professional mores that keep teachers from 
acknowledging weaknesses and trying to collectively to remediate poor teaching? 

» Did the Academy get good support from principals of participating schools? 

• How well did Academy teaching reform strategy deal with teacher turnover? 



Evaluation Report Topics/Issues 

• The burden on teachers of providing them assistance. 

• Lack of agreement on goals for mathematics and science. 

• Misdirection of strategic planning. 

• The insuitability of science teaching packages. 



